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ABSTRACT: As a further step towards maturity, the field of learning analytics (LA) is working 
on the definition of frameworks that structure the legal and ethical issues that scholars and 
practitioners must take into account when planning and applying LA solutions to their 
learning contexts. However, current efforts in this direction tend to be focused on 
institutional higher education approaches. This paper reflects on the need to extend these 
ethical frameworks to cover other approaches to LA; more concretely, small-scale classroom- 
oriented approaches that aim to support teachers in their practice. This reflection is based 
on three studies where we applied our teacher-led learning analytics approach in higher 
education and primary school contexts. We describe the ethical issues that emerged in these 
learning scenarios, and discuss them according to three dimensions: the overall learning 
analytics approach, the particular solution to learning analytics adopted, and the educational 
contexts where the analytics are applied. We see this effort as a first step towards the wider 
objective of providing a more comprehensive and adapted ethical framework to learning 
analytics that is able to address the needs of different learning analytics approaches and 
educational contexts. 
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1 INTRODUCTION 

The increasing trend towards massive data collection in educational settings has raised new ethical 
concerns in the learning analytics (LA) research community. On the one hand, there is a need to 
identify students across platforms and retrieve as much data as possible in order to obtain informed 
analysis about the learning processes. On the other hand, other aspects influence the adoption and 
acceptability of learning analytics approaches, such as data ownership and openness, potential 
abuse and the need for new key competences to interpret and act on learning analytics results 
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(Greller & Drachsler, 2012). These and other aspects are part of the ethical concerns that the field of 
learning analytics must face in order to grow as a mature discipline. 

Several authors have reflected on the ethical issues that affect the field (Slade & Prinsloo, 2013; 
Sclater, 2014) and have made proposals to deal with them, like the set of design guidelines proposed 
by Pardo and Siemens (2014). However, most of these analyses and proposals apply to higher 
education institutional contexts. Little reflection has been done on the implications of using learning 
analytics in smaller-scale contexts where teachers use the data to manage their university or school- 
level classrooms, especially on the latter. However, as pointed out by Griffiths (2012), the kind of 
ethical considerations that must be taken into account are different depending on the approach 
taken to learning analytics. Therefore, there is a need to reflect on what ethical aspects are relevant 
in the applications of learning analytics to small-scale teaching practices, and especially in school 
contexts. 

During recent years, we have worked on the support of teachers' orchestration (Dillenbourg, 2013) 
of CSCL activities in blended learning scenarios. With the aim of providing teachers with meaningful 
and easy-to-appropriate data, we have proposed both scripting and monitoring processes, each 
aware of the other, so that the analysis and results provided by the learning analytics system are 
influenced by the information provided (by the teacher) at design time (Rodrfguez-Triana, Martfnez- 
Mones, Asensio-Perez, & Dimitriadis, 2012; Rodrfguez-Triana, Martfnez-Mones, Asensio-Perez, & 
Dimitriadis, 2013). Both processes are supported by a technological infrastructure — a data 
integration architecture — able to integrate different kinds of data sources, including LMS, tools, 
and user-generated data (Rodrfguez-Triana, Martfnez-Mones, & Asensio-Perez, 2011). This 
integration aims to apply learning analytics to the existing technological ecosystems with which the 
users (teachers and students) are familiar. 

These proposals were designed and validated following a design-based research (DBR) process 
applied to seven cases in higher education (Rodrfguez-Triana, Martfnez-Mones, Asensio-Perez, & 
Dimitriadis, 2015). To test the capability of our solutions to adapt to different educational levels, we 
applied them to a primary school case. We found that while we could easily manage the technical 
aspects of data retrieval, analysis, and visualization, emergent issues related to data ownership and 
control, student identity, and other legal and ethical concerns were more difficult to handle. 

We present, in this paper, a reflection on how the ethical frameworks proposed in LA literature 
apply to our teacher-centred classroom-based approach and to the two aforementioned educational 
contexts. The two studies presented in the paper illustrate with concrete examples which ethical 
issues are relevant in learning analytics when applied to small-scale teacher-led innovation in higher 
education and at school levels. The findings of the studies provide initial evidence of the need to 
adapt existing ethical frameworks to the different approaches to learning analytics and to the 
educational contexts to which they may be applied. 
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The rest of the paper is structured as follows: Section 2 provides an overview of the ethical issues 
identified in the learning analytics literature; Section 3 describes the main aspects of our approach 
to LA; Sections 4 and 5 explain how we applied our learning analytics approach in university and 
primary school contexts as well as the main ethical issues that we encountered; Section 6 discusses 
and compares the findings obtained from the different contexts; and finally, Section 7 provides the 
main conclusions drawn from this work. 

2 ETHICAL ISSUES IN LEARNING ANALYTICS: A FRAMEWORK OF 
REFERENCE 

As the field of learning analytics emerges from its infancy, ethical issues related to its application to 
real practice are receiving more attention. In parallel with the increasing impact of the proposals 
coming from the field, there is a need to reflect on the consequences that these proposals have on 
the persons involved (Slade & Prinsloo, 2013), and on how learning analytics must be shaped to 
respond to these demands (Pardo & Siemens, 2014). 

We aim to provide new insights on how ethical issues depend on the approach to LA, and on the 
particular contexts to which these approaches are applied. In order to structure the analysis, there is 
a need to organize the issues and establish a common framework that enables discussion about 
them (Ferguson, 2012). However, to the best of our knowledge, few frameworks yet address the 
ethical issues related to the application of small-scale teacher-led learning analytics in schools. There 
exist studies that draw attention to the potential benefits of learning analytics for primary schools 
(see e.g., Ebner & Schon, 2013), or deal with how to support LA-enabled teacher interventions 
(Wise, 2014), but they do not usually address the ethical or privacy concerns that may affect them. 

The existing frameworks take an institutional approach. Kay, Korn, and Oppenheimer (2012) identify 
a set of legal and ethical issues, with a focus on higher education. Sclater (2014) provides a 
comprehensive list of concepts and issues related to ethics, accompanied by an extensive literature 
review. Based on this review, and after a process of structured discussion and validation involving 
experts and stakeholders (Sclater, 2015), a Code of Practice for learning analytics was released to 
support educational institutions in the effective use of learning analytics (Sclater & Bailey, 2015). In 
the rest of this section, we describe the categories defined in the Code of Practice, pointing out how 
the issues covered apply to the teacher-led small-scale learning analytics approach addressed in this 
paper. 

Responsibility: There is a need to define who is responsible for the legal, ethical, and effective use of 
LA. Concretely, specific responsibility should be allocated for data collection, anonymization, 
analysis, retention, and stewardship, as well as intervention. At the classroom level, teachers will be, 
in many cases, the ones responsible for these aspects. One question derived from this is whether the 
teachers involved have the capacity if provided with the tools needed to face this responsibility. 
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Transparency: This aspect refers to how and whether students should be informed about data 
collection, results, and the methods of analysis. Any potential adverse consequences of withdrawal 
from a learning analytics process should also be explained to them. At the institutional level, it 
makes sense to ask whether teachers should be aware of this information, while in small-scale 
approaches, this question does not normally apply, as the teachers will be the ones to control the 
analysis. Therefore, the main concern in this case is whether and how the envisioned data collection 
and analysis are to be presented to the students. 

Consent: This topic refers to how and in which circumstances students (parents) should be asked for 
consent to collect and use their data, e.g., aspects related to informed consent, whether students 
hold the right to opt out, and what the consequences of doing this will be for the individual and for 
the group. Consent can be formulated to enable students to decide on the data being collected 
about them, or to preserve their anonymity. This topic also includes issues related to the informed 
nature of the consent, asking whether consent can be considered as informed when the participants 
do not understand the analytics, or when these analytics may evolve in the future in unknown ways. 
The need to describe the analysis so that the participants understand it may be very relevant at the 
school level, where the participants (teachers, children, and their parents) may find it difficult to 
understand what is planned. At the classroom level, opting out may not be possible if the analysis 
forms part of the learning activities. In cases where it is possible, one issue to address is how to 
provide teachers with logistical support to exclude these learners from the analysis. 

Privacy: Access to student data and analytics must be restricted to those with a legitimate need to 
view them. There are important issues related to the capacity of learning analytics algorithms to re¬ 
identify individuals based on the aggregation of these data sources. If there is a need to use 
"sensitive data" (e.g., ethnicity, religion, sexual preference) additional safeguards and possibly 
additional consent should be collected before granting any request from an external body to share 
data. Privacy may be less important at the classroom level, when data is only used and shared within 
the class. However, it is necessary to ensure that privacy is not being violated, especially when 
working with minors, where this is a very sensitive issue. 

Validity: This concept refers to the extent to which the analysis can be trusted. Relevant questions 
relate to which data sources are necessary to ensure accuracy, how to verify the algorithms used to 
draw conclusions, and how to avoid drawing conclusions from spurious correlations. In contexts 
where learning analytics is applied to historical data, it is also necessary to consider that students 
develop, learn, and grow, and the conclusions based on that data must consider that development. 
Finally, this aspect refers to a more technical issue related to the authentication of data sources 
coming from public sites or, in general, from third parties, i.e., how to ensure that students are 
correctly identified when using data coming from these external data sources. In classroom-based 
experiences, where blended learning is common, there are many threats to validity, as participants 
may interact in many forms, including face to face outside the classroom. 
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Access: This topic relates to whether and in which ways students can access the analytics that make 
use of their data. This aspect considers the need to allow students to correct inaccurate data about 
themselves. At the small scale, teachers can ask the learners or consider extra data sources in order 
to make corrections. 

Enabling positive intervention: This topic relates to what should be done by the institution with the 
information obtained, and what the consequences of not doing anything are, e.g., not informing a 
student that s/he risks failing a subject. This aspect also includes issues related to pedagogical 
interventions, such as who should be responsible for presenting the results and how, the possible 
danger of favouring one group over another, how to adapt the intervention to the different needs of 
the students, et cetera. The approach discussed in this paper assumes that learning analytics is 
applied to support teacher interventions in the class. As with any other pedagogical intervention, it 
is necessary to reflect on the impact that these interventions (or lack thereof) have on the learners. 

Minimizing impact: This topic covers some of the main adverse impacts already identified in the 
application of learning analytics and how to deal with them. We should be aware that an analysis 
can never give a complete picture of an individual's learning and may sometimes ignore personal 
circumstances. Thus, we must ensure that trends, norms, categorization, or any labelling of students 
does not create bias, reinforce discriminatory attitudes, or increase social exclusion. At the small 
scale, this aspect is closely related to the effect that the intervention will have on the students. 

Stewardship of data: This aspect is related to the administration of data by the institution, including 
compliance with the legal requirements applicable in each case. Data for learning analytics must 
comply with existing institutional data policies, and these policies must ensure that the rights of all 
the involved persons are maintained. Additionally, only the minimum data required to deliver the 
purposes of the analytics should be collected and they should only be retained for appropriate and 
clearly defined periods. Teachers require help in accomplishing these tasks. 

This classification, proposed by the Code of Practice, was useful to structure the discussion about 
ethical concerns found in the cases described in Sections 4 and 5. Moreover, this discussion helped 
us identify aspects where the framework should be adapted to the approach to learning analytics 
addressed in this paper. The next section outlines the main characteristics of this approach. 

3 OUR APPROACH TO LEARNING ANALYTICS 

Griffiths (2012) identifies two types of analytic interventions with potential impact on teaching 
practices: 1) methods oriented to achieve enhanced regulation of the teaching and learning 
environment; and 2) methods and tools intended to help lecturers carry out their tasks more 
effectively. Our approach to learning analytics is aligned with the second approach. It is devoted to 
supporting teacher reflection, as opposed to prediction, to use the terms of Greller & Drachsler 
(2012). This section outlines the main components of our approach, highlighting those aspects of the 

ISSN 1929-7750 (online). The Journal of Learning Analytics works under a Creative Commons License, Attribution - NonCommercial-NoDerivs 3.0 Unported (CC BY-NC-ND 3.0) a -j 


JOURNAL OF LEARNING ANALYTICS 


S8LAR 

SOCIETY for LEARNING 
ANALYTICS RESEARCH 


(2016). Learning analytics in small-scale teacher-led innovations: Ethical and data privacy issues. Journal of Learning Analytics, 3(1), 43-65. 
http://dx.doi.Org/10.18608/jla.2016.31.4 


proposed solutions with an impact on the ethical issues discussed in this paper. 

Our learning analytics approach is designed to provide monitoring information to be used for 
regulation, formative assessment, or self-reflection about the learning design and the learning 
process. We aim to provide teachers with feedback about the accomplishment of pedagogical 
decisions made at design time. The educational settings where the approach can be applied cover 
face-to-face and distance activities carried out at different social levels (individual, group, and whole 
class). In other words, blended Computer-Supported Collaborative Learning (CSCL). In addition, in 
terms of technological support, the proposal is devoted to distributed learning environments (DLEs) 
made up of an institutional virtual learning environment (VLE), Web 2.0 tools, and GLUE! — an 
architecture used to integrate third-party tools into VLEs. As described in the rest of this section, our 
work was designed to address specific challenges met in these contexts that also have ethical 
implications. These challenges included the need to support teachers in taking responsibility for data 
analysis and interpretation; how to help them with stewardship of data in these complex 
technological systems; as well as how to enhance data validity in blended learning supported by 
DLEs. 



Monitoring-aware 



SUMPSE^ 


Figure 1: Overview of the main components involved in our learning analytics approach. 

The solution consists of three components (See Figure 1): two processes and an architecture for data 
gathering and integration in DLEs to help teachers integrate learning analytics within their practice. 
The first component is a monitoring-aware design process of the learning scenario that takes into 
account the teacher's information needs (Rodriguez-Triana et al., 2013). This enriched design 
process helps teachers identify and make explicit which moments or aspects of their envisioned 
learning activities should be monitored. These aspects are based on the identification of a set of 
constraints: special conditions that must be met to comply with the pedagogical intentions 
expressed in the design, e.g., whether participation of all the students in a group is required in a 
certain phase of the activity, or whether a particular product must be delivered at a specific 
deadline, et cetera. The process entails a second phase, during which teachers are prompted to 
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enrich the design to satisfy these monitoring needs (e.g., choosing the most appropriate tools for 
both pedagogical and monitoring concerns, or identifying complementary data sources). The second 
component of the solution is a monitoring process guided by the decisions made at design time 
(Rodriguez-Triana et a I2012). In this monitoring process, the data gathering is focused on those 
sources chosen by the teacher and the data analysis is designed to verify whether the current state 
(the gathered evidence) matches the desired state (the learning design). These two processes 
(design and monitoring) are two sides of an overall approach that aims to help teachers take control 
of the data analysis and interpretation. By means of the design process, they become active 
participants in the definition of the analysis. The contextualized visualization of the results enabled 
by the script-aware monitoring process is designed to support teachers in their interpretation of the 
analyses. 

The third component of our proposal is GLUEI-CAS, an architecture that addresses the need of data 
gathering and integration in DLEs made up of VLEs (typically Moodle or Mediawiki) and Web 2.0 
tools (e.g., Google applications) (Rodriguez-Triana et al., 2011). GLUEI-CAS defines how to collect 
and integrate data coming from these data sources, overcoming the problems of gathering data 
from external tools when using a VLE. However, in blended scenarios, these automatic data sources 
are not enough to get a full picture of the interactions taking place. Frequently, part of the learning 
process occurs outside the technological context. Besides, the information and communication 
technologies (ICTs) register a limited set of evidence, usually based on user interactions with the 
platforms, and sometimes it is not possible to authenticate the student identity properly (Slade & 
Prinsloo, 2013). Since restricting the analysis to the data registered by the ICT tools may offer only a 
partial view of the user activity (Avouris, Fiotakis, Kahrimanis, Margaritis, & Komis, 2007), GLUEI-CAS 
enables the integration of ad-hoc information provided by the participants in the learning context. 
This practice enriches the evidence gathered and allows teachers to triangulate the data coming 
from the different data sources. From an ethical point of view, this approach enables teachers and 
students to correct the data automatically collected from the ICT tools (Sclater, 2014) and to better 
understand the reasons behind the results obtained (Greller & Drachsler, 2012). Overall, this 
integration of heterogeneous data sources is designed to increase the validity of the analysis. In 
terms of privacy, this involvement of the stakeholders also has benefits, letting them decide what 
information they want to share (Slade & Prinsloo, 2013). Finally, the technical solution helps 
teachers to face the difficulties associated with data stewardship in DLEs. 

To support teachers in their analysis, we implemented GLIMPSE (Rodriguez-Triana et al., 2013), a 
tool that, interacting with GLUEI-CAS, automates data gathering, integration, and analysis, offering 
the teacher a comparison between the current and the desired state of the learning scenario. The 
product of this tool is a monitoring report where the information is visualized by means of tables 
that connect the participants, the data sources, the indicators, and the warnings that emerged from 
the analysis (see Figure 2 below for an example). 
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4 FIRST SCENARIO: HIGHER EDUCATION 

The proposal presented in the previous section was iteratively applied to seven learning scenarios in 
higher education (Rodrfguez-Triana et al., 2015). In this paper, we focus on the last two of these 
scenarios, which were designed to evaluate the proposal in its final version. This section offers an 
overview of the learning contexts as well as how the proposal was implemented throughout the 
learning scenario lifecycle. We will not describe the results related to the application of the learning 
analytics approach as such, which have been reported elsewhere (see Rodrfguez-Triana et al., 2015). 
We will focus on illustrating those aspects of the cases that had an influence on the ethical issues 
described in Section 2. 

4.1 Learning Context 

The approach described in Section 3 was applied to two learning scenarios in higher education with 
a common profile: 3-4 weeks, implementing learning designs inspired by CSCL principles and 
supported by DLEs, interweaving face-to-face and distance activities as well as blended interactions 
among students. The studies involved two teachers from different backgrounds, with different levels 
of expertise in CSCL scenarios, and different knowledge about the proposal. To reference these 
studies, we will use HE1 and HE2 as labels for the first and second scenarios in higher education. 

The first study (HE1) was carried out in an Early Childhood Education course, involving a teacher who 
was not expert in CSCL scenarios and who had less than 6 years of teaching experience. Of the 165 
students enrolled in the course, 150 participated. The learning scenario lasted 4 weeks and consisted 
of various distance and face-to-face activities combining individual and collaborative work. The 
whole learning design was supported by Moodle and Google applications, and involved a total of 
316 resources. The main challenge of this scenario was to cope with the high number of students 
and resources. 

The second study (HE2) took place in an educational research course that formed part of a Master's 
degree for pre-service secondary education teachers. The teacher in charge of this course was an 
expert in CSCL and had previous knowledge of the approach. Over 3 weeks, 15 students were 
involved in this study. They worked on defining a proposal for an educational research project, 
combining individual, group, and class activities, as well as face-to-face and distance learning. The 
whole learning process was technologically supported by means of MediaWiki and Google 
applications, requiring 77 files in total. The main challenge of this scenario was the complexity of the 
design: many interrelated activities occurring in a short period of time that demanded much 
attention from the teacher to avoid problems that could jeopardize the scenario. 

4.2 Application of the Proposal 

The application of the proposal started with the design of the learning scenarios, following the 
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monitoring-aware design process (see Section 3). This process is designed to give responsibility of 
the learning analytics process to the teacher and provides a means of addressing the validity of the 
analysis. Besides paying attention to the pedagogical aspects of the learning activities, the design 
process supported the teachers' articulation of their monitoring concerns. Based on the information 
provided in the design, and following the proposed design process, the teachers were informed 
about the data available to verify the constraints of the envisioned learning scenario, whether there 
was enough evidence to evaluate them, and which complementary data sources could be used to 
improve the analysis. This information gave the teachers the opportunity to reflect on the 
monitoring process and enhance it. In both scenarios, the teachers decided to triangulate the data 
coming from the ICT tools with data provided ad-hoc by the students and the teachers themselves. 
These aspects are illustrated in the reflections made by both teachers when they were interviewed 
after the design phase: 

It has helped me to first reflect about which were the most conflictive points of the design. 

Once they were identified, and knowing which technological tools I was going to use in each 
phase, I could know both what kind of information they would provide and what the best 
moment to get that information would be. [...] In the original design, I did not foresee 
controlling classroom attendance in a systematic manner, nor administering a questionnaire 
to find out the collaboration, task distribution, and workload aspects for each of the phases. 

[...] The design process made me appreciate the importance of including new information 
sources to enable gathering additional evidence. [HE1, Teacher interview after the design 
phase] 

In case the tools that I had in mind did not provide monitoring information, I would have 
substituted them with other tools (provided they have similar functionality to support the 
students' work, of course). [HE2, Teacher interview after the design phase] 

Thus, throughout the monitoring-aware design process, the teachers were responsible for the data 
gathering and analysis, defining which constraints needed to be evaluated during the enactment, 
selecting the data sources, the information to be retrieved, and when it should be collected. 

A second ethical aspect addressed in the proposal regards transparency and consent. Once the 
teachers defined the analysis to be carried out, the students were informed about the data to be 
gathered, the metrics, the purpose of the analysis, and its potential impact. We then asked them to 
give their permission for data collection to take place: almost all the students participating in the 
learning scenarios agreed to this (150 out of 165 in HE1 and all 15 in HE2). The tools supporting the 
design process allowed us to specify which students were taken into consideration, focusing the 
analysis on those students who had given their consent. Therefore, our proposal was able to 
address, at least from a technical point of view, the problems associated with letting learners opt 
out from the analysis. 

A third issue relates to the stewardship of data. During the scenarios, the students' data regarding 
the actions registered on the different platforms (e.g., accesses, editions, uploads, et cetera) was 
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gathered from the technological environment. The data came from the VLEs (Moodle and 
MediaWiki), the Web 2.0 tools (e.g., Google Apps), and the architecture that supported the 
integration of the tools in the VLEs (GLUE!). On the one hand, both the VLEs and GLUE! were hosted 
on our own servers. Thus, we did not have to deal with permission issues in order to connect the 
data gathering tool (GLUEI-CAS) with these platforms, which normally constitutes an obstacle when 
working with institutional or third-party platforms. On the other hand, GLUEI-CAS allowed us to 
collect the data from the third-party Web 2.0 tools integrated in the DLE. Thus, even if the 
information was initially controlled by the owners of these tools, we made it available for the 
teachers to use for their analyses. The integration of these data by GLUEI-CAS enabled teachers to 
control these data, and use it even when it was removed or no longer available in the original tools. 

The integration of data in GLUEI-CAS is also important to increase the validity of the analysis. As 
mentioned above, this focus on validity is addressed by the design process, which promotes teacher 
reflection on the accuracy of the data and enables them to include new data sources to enhance this 
validity. Following these design decisions, teachers and students contributed actively by providing 
their own data. The teachers provided information about student attendance at the face-to-face 
sessions together with a few notes, and the students, by means of questionnaires, described how 
they had participated in the learning activities (how they had interacted, which tools were used, how 
much time they had devoted to the tasks, et cetera). 

Via GLUEI-CAS and GLIMPSE, data generated by the multiple data sources were collected, 
integrated, and analyzed according to the teacher's decisions. Based on the data gathered, the 
teachers received the corresponding monitoring reports taking into account the relevant data 
sources, the actions to be logged, the timeframes, and the indicators to be verified. Error! Reference 
source not found, shows one of these monitoring reports sent to the teacher (only three groups 
appear due to constraints of space). The columns show, from left to right: the groups and their 
members, the data sources employed in this case, and the warnings issued by GLIMPSE when a 
condition specified at design time is not met. The cells coloured in green refer to students or 
activities that have complied with the specified constraints, while the red cells highlight the points 
where there is no evidence supporting the teacher's expectations about the students' involvement 
or the use of resources. For example, Figure 2 shows that no access by StudentName6 to the Final 
research proposal had been registered. The teacher had specified at design time that this resource 
had to be used at least once by each group member, as the lack of access could imply a lack of 
participation of that member in the authorship of the report. This problem was visualized in the 
report by means of the red cell corresponding to StudentName6, and the warning that appears on 
the right-hand side of the table. As can be observed, the reports summarized the evidence collected, 
highlighting potential problems that needed to be checked. 
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Figure 2: Excerpt from the monitoring report sent to the teacher the HE2 study (anonymized 

version). 

The integration of multiple data sources in the analysis played a crucial role in the success of the 
proposal. First, it significantly increased the accuracy of the results. Complementing the evidence 
from the ICT tools with the teacher's observations and the students' feedback allowed us to take 
into account not only the computer-mediated interactions but also the face-to-face ones. 


The students' report has been very useful, in order to take into account their perspective. 

[...] Being able to incorporate my notes with the comments received from students has 
simplified my work a great deal. [...] Having all the data gathered in one place simplifies the 
monitoring. Having the information centralized helps avoid misunderstandings and keeps the 
situation under control. [HE1, Teacher interview after the enactment] 

The integration is not only useful, it is necessary. It gives you a complementary view of an 
activity that happens in different settings/moments. It uses blended sources to inform 
blended activities. [HE2, Teacher interview after the enactment] 

We evaluated the validity of the monitoring reports by comparing the results obtained with the 
complementary teacher observations, the additional student comments, the researcher 
observations, and the learning outcomes in the tools. Despite the existence of errors, the monitoring 
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reports provided a perspective of the learning process that was close to the real facts (in 99.67% and 
97.81% of the evaluated conditions). 

As previously discussed, our approach hands responsibility for the analysis (both the design and the 
interpretation) to teachers. One aspect that must be addressed regarding responsibility is whether 
teachers are able to appropriate the analyses and act upon them. As can be seen in Figure 2, the 
information provided to teachers was direct and contextualized in their designs. Both teachers 
agreed that interpreting the reports could be done quickly and did not present any problems. 

Interpreting the monitoring reports was very easy. With one look, I knew whether there had 
been any problems. [...] I dedicated 10 minutes at most: 5 minutes to read everything, plus 
another 5 minutes to take the corresponding measures. [HE1, Teacher interview after the 
enactment] 

Interpreting the reports was simple and immediate. The information provided is clear and 
does not lead to misinterpretations. [HE2, Teacher interview after the enactment] 

During the learning scenario, the teachers used the information to identify potential problems as 
well as to intervene and regulate the scenario when appropriate. Afterwards, once the activities 
ended, the teachers employed the monitoring reports to support reflection on the learning process. 

When I detected a problem, I contacted the student. In fact, I have sent quite a few emails 
with wake up calls, reminders, and asking what happened. This is something we normally do 
not do because we do not have means to follow the students' work closely. [HE1, Teacher 
interview after the enactment] 

The monitoring reports required regulatory tasks only in two cases. In general, all 
students/groups followed the plan, except for the development of the report in small groups 
(which I commented on verbally with them in the classroom) and in the last activity, in which 
three students forgot to send the peer-assessment report, and thus I had to send a reminder 
to them via email. It was critical because neither they nor I would have remembered until 
the final assessment, which coincided with the end of the course. The rest served to check 
that everything was going well. [HE2, Teacher interview after the enactment] 

Regarding student access to the data analysis, we should remember that our approach is devoted to 
supporting teachers. Thus, during the scenarios students did not automatically have access to the 
monitoring reports. However, the messages sent to the teacher in the reports made it clear that the 
results had to be checked. Therefore, any non-expected result should have been discussed with the 
student, who could clarify the reasons behind it. The teachers' comments during the interviews 
presented earlier illustrate how they proceeded in such cases. 

In summary, the two university cases illustrate how our classroom-based learning analytics approach 
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provides teachers with tools to become responsible for data analysis; how data stewardship was 
solved by the use of GLUEI-CAS (and the fact that we could use our own servers at the university); 
how we dealt with the need to provide transparent information to the participants; and how the 
technical solution enabled teachers to exclude from the analysis those who did not sign the consent 
form. We have also shown our emphasis on the validity of the analysis by means of the integration 
of several data sources, planned at design time and addressed technically by GLUEI-CAS; and how 
the results were shown by GLIMPSE in a contextualized manner that teachers could appropriate to 
intervene in the learning situations when needed. The students did not receive the data directly, but 
teachers were prompted to compare results with other available information before acting, as the 
whole proposal makes teachers aware of the weaknesses associated with data analysis in blended 
scenarios. 

5 SECOND SCENARIO: PRIMARY SCHOOL 

As we described in Section 3, our proposal was designed to help teachers in blended CSCL scenarios 
supported by DLEs. Although all our previous studies had been carried out in higher education, we 
realized that at other educational levels, teachers faced similar problems. Thus, we set out to apply 
our proposal in a primary education setting with the aim of checking its adaptability to this new 
context, and to unveil the potential limitations and constraints imposed by such a learning context. 
This section provides a brief description of the learning context including an analysis of the different 
ethical issues we faced during the study. 

5.1 Learning Context 

The case study was run in a first grade class (6-7 years old) with 24 students at a rural school in 
Valladolid, Spain. This school has 300 students and 29 teachers. The teacher involved in this study 
used Blogger in his classes in combination with external Web 2.0 tools (such as YouTube videos or 
Educaplay games). In spite of a general positive experience with this approach, one of his main 
concerns was that these blogs did not tell him who had read or accessed the pages. This precluded 
him from knowing whether the students were using the learning materials offered in the blog. 

In order to address this problem, we used GLUE! to build the DLE. Blogger was treated as the VLE 
where the third-party tools (e.g., the videos and games) were embedded. With this technological 
setting, it was possible to trace student activity during the learning process and then inform the 
teacher about their use of the tool. 

The intervention ran between 14 May and 2 June 2014. The teacher proposed two lesson plans 
involving a blog and several external resources, in which the teacher asked the students to carry out 
some activities at home such as watching a video, reading an online text, or playing educational 
games. 
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5.2 Application of the Proposal 

As with the previous study, during this design process, the teacher described the learning activities, 
chose the resources to be used, and configured the ways in which the learning scenario would be 
monitored: he decided which constraints should be checked, specified student actions to be 
monitored, as well as the timeframes and delivery dates of the monitoring reports. 

Once the teacher configured the analyses, we informed the different participants, namely the school 
principal and the parents/legal guardians of the students, since they were not adults. A detailed 
explanation of the data, metrics, purpose, and potential impact of the analyses was given to them. 
Both the school principal and the families supported this kind of innovative practice and gave their 
consent for the data collection. 

Regarding the collection of identified data, we encountered an unexpected legal obstacle. Blogger 
can be configured so that only registered users access the blogs, but this requires a Gmail account. 
However, 6-7 year old children are not legally allowed to own an email account on that server. In 
order to overcome this obstacle, we had to look for a workaround, asking the families to open an 
email account on behalf of their children, and then access the site using that account. Even so, we 
did not find it difficult to obtain consent and family support to create and use the dummy e-mail 
accounts. However, we must note that obtaining consent when working with minors is a very 
delicate issue and must be carefully addressed. 

Another issue we encountered related to the kind of data that could be collected and analyzed. In 
upper educational levels, the input and interactions with technologies may be very rich. However, at 
the age of 6 to 7, students frequently do not have the skills to carry out very complex tasks (e.g., the 
students wrote little before the end of the course). Thus, the kind of data available is likely to be 
limited, and many online educational tools do not offer any data about the learner's work. In fact, 
the data gathered from the technological environment mainly consisted of the actions logged by 
GLUE!, specifically student accesses to the Web 2.0 tools. This information was easily accessible 
because GLUE! was hosted on our servers and we could access the systems without additional 
bureaucracy. 

While the learning scenario was taking place, the teacher received monitoring reports according to 
the plan drawn up at design time. Figure 3 shows a piece of information from those monitoring 
reports. These reports simply presented the evidence collected (accesses to the tools) highlighting 
potential problems that needed to be checked, e.g., students who had not accessed the tools, and 
therefore, could not have used the resources. The monitoring report presented in the figure refers 
to the number of times the students accessed the activities in one of the monitored lesson plans. It 
can be seen that only three students of 24 completed all the activities, while nine out of 24 did not 
access any. The same kind of visualization was shown to the teacher for the second lesson plan. The 
results relating to the participation of the students were similar. Only five students out of 24 
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completed all the activities, while 11 out of 24 did not access any. When the researchers shared 
these results with the teacher, he felt confused, as we can see in the following extract: 

Honestly, I did not think this could happen. I mean, I thought that the majority of my 
students could complete the activities. I think it is important to communicate these results to 
the families. [PE, Teacher interview after the enactment] 
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Figure 3: Output shown to the teacher about the number of students who accessed the activities 

in Lesson Plan 1. 


During the face-to-face sessions, the teacher asked the students whether they had done their 
homework in order to compare their answers with the monitoring reports. The information was 
generally accurate and helped the teacher to intervene appropriately. However, one of the issues 
that we faced related to the students' virtual identities. As noted by the teacher in the second 
interview, not all families knew how to solve some of the technical difficulties, and some families 
accessed the site using the parents' own accounts. 

When I met with the families, many of them told me that they had had many problems 
accessing the activities embedded in the blog. They said that they had read and followed all 
the instructions I gave, but still found it impossible. Moreover, I noticed when I reviewed the 
monitoring report, that there were families that had accessed the activities with a different 
e-mail account. [PE, Teacher interview after the enactment] 

Obviously, the technical difficulties encountered by the families, together with the workaround we 
had to use to be able to identify the children using surrogate accounts, resulted in a case of 
"enmeshed identities" (Greller & Drachsler, 2012). Aware of this problem, in the final interview, the 
teacher said that if he were to use a similar activity in future, he would devote more time to training 
the families to access the on-line activities with the e-mail login, as well as supervising their usage. 


ISSN 1929-7750 (online). The Journal of Learning Analytics works under a Creative Commons License, Attribution - NonCommercial-NoDerivs S.O Unported (CC BY-NC-ND 3.0) 


57 















































JOURNAL OF LEARNING ANALYTICS 


S8LAR 

SOCIETY for LEARNING 
ANALYTICS RESEARCH 


(2016). Learning analytics in small-scale teacher-led innovations: Ethical and data privacy issues. Journal of Learning Analytics, 3(1), 43-65. 
http://dx.doi.Org/10.18608/jla.2016.31.4 


In spite of the simplicity of the project and all the restrictions related to the quality of the data, the 
teacher was very positive about the feedback provided: 

It is important to provide students with other ways of learning, as well as to assess learning. I 
can say that these teaching and learning practices are innovative for students and we have 
the opportunity to motivate them and engage the families in the educational process. [PE, 
Teacher's reflective diary] 

In addition, the teacher identified several potential uses of the analysis. Among other uses, he 
highlighted knowing in advance whether the students had done their homework, being able to send 
reminders, identifying more/less attractive resources, detecting which days of the week the students 
were more active, analyze the students' development in relation to their homework, et cetera. 

Regarding student access and privacy, we should recall that the teacher is the target user of the 
approach presented in this paper. Nevertheless, the results, especially in the case of unexpected 
behaviours, should be checked with students/parents who can clarify what happened. Especially in 
this case, the meaning of the reports might not be easily understandable by children due to their 
age. But the teacher could share it with the parents or with the institution itself as "evidence" of the 
learning process to be discussed. Therefore, teachers are in charge of sharing the data analyses with 
those who legitimately need to view them, and this sharing must take into account the privacy and 
legal issues applicable in each case. 

6 DISCUSSION 

The studies presented in Sections 4 and 5 illuminate our discussion about how the current 
frameworks related to ethical issues in learning analytics apply to the analytical interventions we 
envision, as well as how they depend on the educational contexts where we applied them. We 
identify three levels of discussion: 1) implications related to the learning analytics approach to which 
this work belongs, i.e., small-scale analytics to help teachers manage their classes; 2) implications 
derived from the actual solutions we propose to support this learning analytics approach, and 3) 
issues related to the educational contexts where the solutions are applied. After the discussion at 
these three levels, we drafted a set of guidelines that could be taken into account by teachers 
aiming to conduct learning analytics scenarios of the kind discussed in this paper. 

6.1 Implications Related to the Teacher-Centred Classroom-Based Approach to LA 

As mentioned above, our approach to learning analytics is oriented to supporting teacher reflection 
in order to help them with the management of their classrooms. This is one of the possible 
approaches to learning analytics identified by Griffiths (2012). The other two approaches are 1) 
institutional approaches that seek to increase the efficiency of the institution; and 2) approaches 
associated with the enhanced regulation of the teaching and learning environment. 
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As mentioned in Section 2, the framework used to structure our discussion responds to an 
institution-led approach. This focus can be observed in the kind of questions posed to analyze the 
concepts, and in the responsible stakeholders identified. The questions included in the framework 
focus on what the role of the institution and/or the students in the overall process of learning 
analytics is. Teachers appear only as secondary actors or — marginally — as potential providers of 
data. In contrast, in our approach the teacher plays the role of the institution — in control of the 
data and the analysis — and of the receiver of the information — role-played by the students in the 
reference framework. 

Due to these differences, we had to adapt the questions posed in the framework to better fit our 
approach. This translation was straightforward. However, the need for adaptation could be an 
obstacle for a practitioner who wanted to employ the framework to analyze a learning analytics 
intervention of the kind discussed in this paper. This justifies the need to refine the ethical 
framework to adapt it to the characteristics of the learning analytics approach. 

A first element of refinement would be the inclusion of teachers as responsible stakeholders, and of 
questions related to their role as data providers and receivers. A second element regards the 
emphasis given to different aspects of learning analytics. For example, while privacy issues are 
paramount in an institutional approach, they have different implications in a closed system, where 
the teacher uses already available data and shares it with the students. In fact, in these cases, new 
ethical issues may arise, more related to classroom orchestration, which should also be identified 
and included in the revised framework. 

6.2 Implications of the Design-Aware Learning Analytics Approach 

Our approach to learning analytics is strongly based on the participation of the teacher from the 
beginning of the lifecycle of the learning activity. It takes into account the need to integrate data 
from different sources — including data provided by teachers and students — that offer a 
complementary view to the evidence recorded by the technological infrastructure. This design- 
aware learning analytics approach helps to address a number of ethical issues common in these 
contexts. 

First, the approach has a strong impact on the validity of the data in these blended learning 
scenarios based on the use of DLEs. The integration of different kinds of data by means of the GLUEI- 
CAS architecture, enriched the analysis. The fact that the design process helps teachers reflect on 
the missing data and enables them to introduce data provided by the participants also increased the 
validity. The results of the two cases show that the output given to the teachers, based on this 
combination of data, was accurate in most cases. 

The second issue addressed by our proposal refers to responsibility for the analysis. The approach 
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helps teachers take decisions about the data and the analytics that will be applied to the setting. 
This schema makes teachers fully aware of the purpose of the analysis, and avoids possible negative 
effects of systems that demand new competencies of teachers in order to interpret the results. 

The possibility of adverse impact is minimized because, in our case, the analysis does not entail any 
negative effect on the students. Indeed, the way the approach is presented to teachers makes them 
realize that the information available is not complete, and they are responsible for asking or looking 
for other data sources to complement it. Thus, no decision is made automatically based on the 
results. The analysis only aims to detect potential problems in order to address them as soon as 
possible. To be more precise, it is the teacher who, aware of the information, should verify the 
situation and intervene if necessary. 

6.3 Issues that Depend on the Educational Context 

Our experience in applying learning analytics to two different educational levels (see Sections 4 and 
5) showed us that ethical issues also depend on the kind of learning analytics processes that can be 
expected in these contexts. As highlighted by the cases discussed in this work, we found new legal 
and ethical concerns in the primary school context that we had not met in higher education 
scenarios. 

A first issue affected by this change of context refers to the stewardship of data. In higher education 
settings, it is usual the institution that owns the servers where the learning environments reside and 
the activities take place. Teachers can access these platforms, and count on technical staff to 
configure the analysis. On the contrary, schools often do not have easy access to this kind of learning 
platform, or, if they do, they do not have access to technical staff who can support the analysis 
(Blackwell, Lauricella, Wartella, Robb, & Schomburg, 2013; Lim, Zhao, Tondeur, Chai, & Tsai, 2013). 
Due to these problems, teachers and schools are increasingly relying on cloud and Web 2.0 tools. 
The dependency on these external Web 2.0 tools poses new legal problems and ethical issues, as 
was illustrated in our school case. Students (minors) cannot legally have their own accounts, leading 
to a conflict with the need for learning analytics processes to identify users. One solution, as we 
described, is to rely on the families of the children to obtain a surrogate identity, but this adds more 
complexity to the analysis and new threats to data validity. In the case discussed in this paper, some 
families were not able to help their children to perform the tasks, while others used the parents' 
identities to access activities. 

In both cases, this had an effect on the validity of the data collected and displayed to the teacher. As 
discussed in the previous section, the teacher knew in advance the limitations of the implemented 
approach, and did not take any direct action based exclusively on the monitoring reports. 
Nevertheless, he found the information provided useful, as it gave him a general overview about 
how the students worked at home with the virtual environment that he had set up. 
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When working with lower level courses (like the first graders studied in this paper), children are not 
autonomous in their use of computers, and families must be considered actors to take into account 
at many levels (permission, training, collection of data, et cetera). The inclusion of families requires a 
reflection on how this change affects the other ethical issues discussed in the field. For example, the 
option of showing the results to the students — and therefore to their families — must be balanced 
against issues such as privacy, transparency, and action, as these are bound to be an issue when the 
families are aware that the tasks carried out at home are being analyzed and used by the teachers. 

A final issue relevant for the application of learning analytics in school contexts is the simplicity of 
the interactions between the children — especially first graders — and the system. This poses some 
questions about the kind of inferences that can be drawn from this very basic information. In 
principle, considering the limitations imposed by the data available, a simple learning analytics 
approach showing basic data to the teacher seems more coherent than sophisticated analyses. 
However, future developments in the field could challenge this view. 

Overall, our analysis suggests that the work being done by the learning analytics community to 
define a general framework or code of practice for learning analytics must take into account the 
approach and the educational contexts where learning analytics is applied. Further work on 
analyzing the particular characteristics common to the different approaches and educational 
contexts would help to identify these new aspects and to refine the proposed frameworks. 

6.4 Proposals for an Expanded Ethical Framework for Learning Analytics 

As a conclusion of our reflection, we propose a set of recommendations to be taken into account to 
extend or complement the existing frameworks of ethical issues in learning analytics. As discussed 
above, the existing frameworks are oriented to institutions and therefore apply to large and 
medium-scale institution-led learning analytic approaches. If we consider that learning analytics can 
be applied in small-scale scenarios, there is a need to adapt the existing frameworks to these 
approaches currently coexisting in the learning analytics field. A possible way to address this goal is 
to define different itineraries depending on the approach to learning analytics, so that practitioners 
can focus their view on the questions most important to their approach. A similar approach is taken 
in Kay et al., (2012), defining the different legal and ethical priorities in six use cases related to 
learning analytics in higher education. 

One of these itineraries could be defined for the learning analytics approach discussed in this paper, 
i.e., classroom-based learning analytics oriented to supporting teachers' regulation of their classes. 
In order to provide an initial proposal, we have adapted the questions posed by Sclater's (2015) 
framework to produce a set of guidelines that could be taken into account by teachers willing to 
apply learning analytics from this perspective. These guidelines are presented in Table 1, ordered 
using the categories employed in the current Code of Practice (Sclater & Bailey, 2015). 
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Contrary to the framework proposed by Sclater (2015), these recommendations assume that, in this 
context, teachers will take responsibility for most of the aspects mentioned in Table 1. Therefore, 
teachers must appear as a main stakeholder in the itineraries addressing these learning analytics 
approaches. 

Table 1: List of recommendations for small-scale teacher-led learning analytics. 


Category 

Recommendation 

Consent 

• If information is already being tracked, inform the students (or families) about it, 
otherwise, ask for formal consent/agreement before data can be collected and/or 
analyzed. 

• Be explicit about what you might do with that information and, if possible, agree on 
these possibilities with the students. 

• Provide students with the option to update their digital dossiers and provide extra 
(possibly qualitative) data but triangulate it to verify they do not "game" the system. 

• Data should be deleted when individuals no longer want it to be processed or when it 
is no longer of use for its original purpose, as is the case with any other student data. 

• Reflect on the consequences that opting out of the analysis will have on the 
participants (e.g., lack of feedback due to the lack of analytics) and inform them. 

Transparency 

• Be explicit about which data is collected, how it is interpreted, why, and how it will 
affect the learning process. 

Access 

• Reflect on whether the students should access the data held about them, the analysis 
of the data, the labels attached to them, and if so, in which format this information 
should be provided to them. 

• Consider the possibility of letting students correct the data stored about them. 

Responsibility 

• You are in charge of interpreting and validating the analyses as well as deciding what 
to do based on the analytics (e.g., how to regulate, intervene, et cetera). 

Privacy 

• If some data must be anonymous, be sure it cannot be re-identified by contextual 

information available to the users. 

• If you use data from external sources, (e.g., Web 2.0 tools) be sure you can identify the 
owner correctly, and that other ethical and privacy issues are not put at risk when 
using those sources. 

Validity 

• The evidence obtained may be incomplete. Try to involve students (and families) in 
order to increase its accuracy. 

Stewardship 

• Use the data needed for the analysis, nothing more. 

• Be sure that you comply with the data protection laws applicable in your region or 

country. 

• The data should be preserved, secured, and shared in the same way as any other 

student data. 

Avoiding 

negative 

impact 

• Consider that your analysis may produce unexpected results that prompt you to 
intervene with the student. Which kinds of obligations will influence your actions? 
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The recommendations presented in Table 1 should be considered as a first attempt to structure 
reflection on ethical issues and logistical concerns in small-scale teacher-led learning analytics. They 
should be subject to refinement by their application to other cases, and by public discussion with 
experts in the field. These discussions might take place in the context of recent initiatives promoted 
by various governments (Hylen, 2015) that highlight the creation of policies and sharing practices of 
learning analytics in schools. 

7 CONCLUSION 

Interest in addressing ethical issues in learning analytics is starting to flourish in the form of ethical 
frameworks that guide codes of conduct for practitioners. These frameworks are useful instruments 
to structure the discussion and promote a more mature application of LA. However, current 
frameworks are focused on institutionally oriented higher education learning analytics. There is a 
need to extend and adapt these frameworks to the characteristics of different learning analytics 
approaches and educational contexts. Through two studies in primary and higher education 
contexts, this work has analyzed the ethical and privacy issues according to three different 
dimensions that may affect them: the overall learning analytics approach, the particular solution to 
learning analytics adopted, and the educational contexts where the analytics are applied. 

The reflection presented in this work shows that the issues associated with smaller-scale teacher-led 
learning analytics require explicit consideration of the role of teachers as main actors in the 
application of the learning analytics processes. Aspects such as action and impact have a particular 
dimension, and are closer to general ethical issues related to classroom orchestration, where 
teachers play a crucial role. In this kind of educational context, it is necessary to analyze from an 
ethical point of view what kinds of actions should trigger the learning analytics output, how to 
proceed, and what the impact will be in terms of classroom management, intervention, regulation, 
and assessment. 

We have shown how our approach to learning analytics, involving teachers from the very beginning 
in the design and enactment of the monitoring process helps to overcome issues that can appear in 
these smaller contexts, such as control of the analytics, awareness, et cetera. Teacher participation 
in the design of the monitoring process contributes to the development of "ethics by design" 
(parallel to the idea of "privacy by design" [Schaar, 2010] in the application of learning analytics 
solutions). According to this schema, the teacher is the one who defines the educational purposes of 
the analysis, reflects on the available data sources, improves the validity of the results (by adding 
new evidence from teachers and students), and is aware of the limitations of the results obtained. 

Finally, the application of the approach to two educational contexts shows that reflection on 
learning analytics must take into account the specific ways of working associated with different 
contexts, including legal aspects that apply to the particular case of schools, where work with minors 
poses specific challenges and prompts the inclusion of families as new actors in the framework. The 
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cloud-based tools that are becoming widespread at these educational levels do not deal with the 
information needs of LA and may pose legal and ethical problems related to data ownership and 
virtual identity, which are difficult to solve when working with minors. Not only LA, but also the 
wider technology-enhanced learning community faces a big challenge when providing appropriate 
tools to these educational levels. 

This work is a first step towards the refinement and adaptation of ethical frameworks to the 
different approaches that currently coexist in the learning analytics field; for example, defining 
different itineraries depending on the approach to LA, so that practitioners can focus on the most 
important questions for their approach. 
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